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Executive Summary 



The Massachusetts charter school initiative was established to provide students and parents 
throughout the Commonwealth with greater choice and quality in their public schools. In exchange 
for the freedom to design its own programs, hire its own teachers, and set its own budget, a 
Massachusetts charter school embraces a high level of accountability. This accountability is 
formalized in the charter school accountability system of the Massachusetts Department of 
Education’s Charter School Office. A lynchpin of that system is a school’s Accountability Plan, 
“establishing specific five-year performance objectives to measure the school’s progress and success 
in raising student achievement, establishing a viable organization, and fulfilling the terms of its 
charter.” (603 CMR 1.05(g)) 

A charter school creates an Accountability Plan to articulate to the community and the state what 
goals the school will use to measure its success. These Guidelines for Writing Charter School 
Accountability Plans are intended to give schools guidance on the type of data that provide compelling 
evidence, the desirable stmcture for goals and objectives, and how to most clearly present results. 
With these tools in mind, schools can write rigorous and measurable accountability plans that will 
serve them well as they participate in the charter school accountability process, especially at the time 
of renewal. Accountability Plans are written in a school’s first year of operation and serve as an 
important tool to guide the use of data and policy decisions regarding the school’s program. 

There is a large amount of information in the pages that follow; it is critical that schools understand 
this information in order to create and track strong performance objectives. Writing plans can be a 
difficult and time-intensive task, as can be setting up the systems to track data in a reliable and 
consistent manner. When a good plan is written, however, it can be a powerful tool to express the 
school’s top priorities, to decide how it will measure success in those areas, and to guide a school’s 
allocation of resources. 

Massachusetts charter schools know in advance the three main issues they will need to address to 
demonstrate that they are worthy of renewal: academic success, organizational viability and 
faithfulness to the terms of the charter. Thus, each plan should address each one of these key areas. 
Aside from this general guideline, each plan should reflect in format and substance the specific 
mission and program of the individual school. While there is no one “correct” way to write a plan, 
there are some overarching questions that a school should ask itself when creating its goals. 

1. Is the plan rigorous and realistic? 

For renewal of a public school charter, the Department of Education does not prescribe one set 
of goals for all schools to meet. While schools are subject to all the requirements of the No 
Child Left Behind Act (NCLB), the charter school accountability system allows schools to set 
specific goals that are appropriate to the school’s unique programs and populations in addition 
to those prescribed by NCLB. At the same time, schools need to create goals that are realistic 
and rigorous enough to warrant renewal at the end of their five-year charter. In writing an 
Accountability Plan, it is important to remember that the fundamental goal of the Plan is to 
prepare an argument that provides comprehensive and compelling evidence of success in the 
three areas of renewal. 
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2. Is the plan measurable? 



The Accountability Plan establishes specific five-year performance objectives to measure the 
school’s progress. These performance objectives will be evaluated each year by the school, the 
public, and the Department of Education through the charter school accountability cycle. 
Because of this, the goals must be measurable both internally and externally, avoiding ambiguous 
phrases and unclear measures. 

3. Is the plan measuring outcomes rather than inputs? 

In establishing performance objectives, it is important to measure results, not inputs. Some of 
the most commonly measured and discussed forms of school data involve what a school is 
doing and implementing rather than the quality of the program and the effect it is having on 
student achievement and organizational strength. Strategies, or inputs, that a school uses to 
achieve their ultimate goals can be articulated in the school’s Annual Reports and other 
documents, but the focus of the Accountability Plan should be on outcomes. 

4. Is the plan focused and manageable? 

An Accountability Plan is not designed to describe all of the outcomes a charter school has set 
for itself. It sets objectives for the most critical areas of its performance that will inform a 
decision about whether to renew the school’s charter. Effective measurement and reporting can 
require a significant commitment of time and resources, and even concise goals can yield a 
lengthy Annual Report. A more powerful case is made when schools measure fewer things 
better than many things incompletely or superficially. In addition, schools should write plans in 
a way that is accessible to any reader, avoiding jargon or unnecessarily complicated language or 
measures. Any reader should read the Accountability Plan and understand what a school has set 
out to do. In turn, any reader should read the Annual Report and know whether the school is 
reaching its goals. 

Once a school community has written an Accountability Plan that can answer each of these 
questions in the affirmative, the school is ready to submit its plan to the Charter School Office of 
the Department of Education. A school submits its accountability plan after the first year of a 
school’s charter and then subsequently with each renewal application. Schools report on their 
progress toward meeting their goals in an Annual Report each August 1, as well as in one-day site 
visits conducted by the Charter School Office. In the school’s application for renewal in year five 
the school provides a final report on its performance in relation to the plan’s goals that are then 
corroborated during the school’s Renewal Inspection Visit. 

A charter school Accountability Plan allows a school to set goals that reflect its uniqueness and 
autonomy while giving substance to a school’s commitment to parents and citizens to provide an 
outstanding education to each student. Charter schools accept the challenges that face all public 
schools and embrace a unique and demanding burden of proof in the accountability inherent in a 
five-year charter. The Accountability Plan, focused on results and objectives of achievement and 
performance, formalizes that challenge in a way that is useful for the school community, the charter 
authorizer, and the public. 
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The Massachusetts charter school initiative was established to provide students and parents 
throughout the Commonwealth with greater choice and quality in their public schools. In exchange 
for the freedom to design its own programs, hire its own teachers, and set its own budget, a 
Massachusetts charter school embraces a high level of accountability. Like all public schools, charter 
schools must comply with applicable laws and regulations in order to operate. To earn renewal of 
their charter, however, charter schools also must demonstrate their effectiveness; they are 
responsible for results rather than relying on plans, methods, or intentions. Charter schools that 
cannot demonstrate the achievement of their students and the effectiveness of their programs face 
non-renewal. 

In addition, charter schools are highly accountable to the public because children attend the school 
at their parents’ discretion. Parents are not required to enroll their child in a charter school, or keep 
them there, if the school is not effective. In short, charter schools must demonstrate their 
effectiveness to both public officials and individual parents in order to survive. As a result, the 

ability to provide effective, compelling evidence of academic and 
organizational performance is central to charter school success. 

Providing evidence of success, therefore, is the purpose of charter 
school Accountability Plan. Each charter school writes an 
Accountability Plan outlining the array of data the school will 
collect and present in proving its effectiveness and meeting its 
burden of accountability. When finalized, the Plan becomes part of 
the school’s “charter” - the agreement between the charter authorizer and the school that allows for 
the school’s existence. Thus, the Accountability Plan is a central element in the bargain of freedom 
to design unique programs in exchange for greater accountability to the public. 

Why Guidelines ? 

Charter schools are given significant leeway to frame the 
argument for the success of their programs; thus. 

Accountability Plans are unique to each school. The 
Guidelines for Writing Charter School Accountability Plans give 
schools guidance on the type of data that provide 
compelling evidence, the desirable structure for goals and 
objectives, and the clearest presentation of results. The 
Guidelines discuss effective practices in educational 
measurement and the ways in which charter schools can 
apply those practices to present the strongest possible case regarding their success. 

The Accountability Cycle and Accountability Plans 

The charter school accountability process in Massachusetts is guided by three areas of inquiry: the 
success of the academic program; the viability of the organization; and the faithfulness to the terms 
of the charter. Because schools must present affirmative evidence of success in these three areas, 
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The Guidelines give schools 
guidance on the type of data 
that provide compelling 
evidence, the desirable 
structure for goals and 
measures, and the clearest 
vresentation of results. 



The ability to provide 
effective, compelling 
evidence of academic 
and organizational 
performance is central to 
charter school success. 





these areas should provide the organizational structure for the Accountability Plan. In writing an 
Accountability Plan, each school should remember that the fundamental goal of the Plan is to 
prepare an argument that will provide comprehensive and compelling evidence of success in the 
three areas of renewal. A well-written and rigorously tracked Accountability Plan provides the 

affirmative evidence for a successful school that supports a strong 
recommendation for renewal. 

A prerequisite to understanding an effective Accountability Plan is 
understanding how this document fits into the larger system of 
charter school oversight. The Accountability Plan is a critical piece in 
the submissions, visits, and evaluations that comprise the 
“Accountability Cycle” 1 and ultimately serves as the basis for renewal 
decisions. 

The first element of the Accountability Cycle is the granting of a 
charter. The charter application serves as a blueprint for each school and outlines the standards to 
which each school aspires. By August 1 after its first year of operation, each school must turn 
general statements of goals included in the charter into a specific and measurable Accountability 
Plan. The Plan must spell out the goals to which the school commits itself and the measures that 
will be used to determine the degree of a school’s success in reaching those goals. 

By August 1 after every school year, each school must submit to the Charter School Office an 
Annual Report. This report’s main purpose is to discuss the school’s interim progress on each goal 
in the Accountability Plan and present evidence to validate its claims. Therefore, the Accountability 
Plan and the Annual Report are pieces of the same process. The Plan should be written with the 
Annual Reports in mind, just as the Reports should be written with the Plan as a continual guide. 

Evidence outlined in the Accountability Plan and presented in the Annual Report is corroborated 
and augmented by a series of additional external evaluations of each charter school, including site 
visits in years two and three, and a renewal inspection following the submission of an application for 
renewal. These visits are intended to augment the results a school describes in reports, providing 
context to a school’s quantitative measures of progress by adding qualitative detail and anecdotal 
evidence in areas that are difficult to measure. The reports are intended to document the school’s 
progress, present additional data that will inform the renewal process, provide parents with 
descriptive information to use in selecting a school, and provide each school with objective feedback 
it can use to make its programs more effective. These purposes are more readily realized if a school 
has clarified its goals and performance standards in an effective and well-designed Accountability 
Plan. 



A well-written and 
rigorously tracked 
Accountability Plan 
provides the affirmative 
evidence for a 
successful school that 
supports a strong 
recommendation for 
renewal. 



1 For more information about the Accountability Cycle, please read “Massachusetts Charter Schools Accountability 
Handbook” at http://www.doe.mass.edu/charter/pdf/acct_handbook.pdf 
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Organization, Format, and Structure 



r Begin With Your Mission Statement 

Each school should begin its Accountability Plan with its mission statement, which reflects its 
distinctiveness, purpose, and reason for being. An effort to demonstrate a school’s effectiveness in 
an Accountability Plan should flow out of this mission. Highlighting the mission statement helps the 
school focus on key elements and sets clear priorities. Reviewers will use the mission statement and 
the priorities it sets to inform their understanding of the school’s progress in fulfilling its mission. 

^ Include Additional Information Cautiously 

A good mission statement leaves out as least as much as it includes. Some schools may want to say 
more to explain whom the members of the community are and what ties them together. These 
schools may wish to include a brief section in the Accountability Plan that provides additional 
information such as a description of the people involved in the school or its philosophy. 
Alternatively, a school that specializes in working with at risk or non-traditional students might wish 
to include some discussion of students’ background in this section. Such a section, however, is not 
necessary for a strong Accountability Plan. 

^ Use Simplest, Clearest, and Easiest Format 

The Guidelines are intended to set standards for clarity, content, and effective measurement and leave 
most decisions about format to the discretion of schools. A school may write its Accountability 
Plans in any format or style, keeping in mind that one of the guiding principles is readability. The 
main goals of the school and how success toward these goals will be measured must be clear to the 
school community as well as to external reviewers. Required elements include the goals and a 
description of the objectives or measures by which the school will assess its achievement of the 
goals. No other sections are required. Again, simple, everyday language is ideal. 

Goals should be written in precise declarative sentences. For example, an effective Accountability 
Plan for the hypothetical Downtown Academy Charter School might list five goals in the Academic 
Program section, each one corresponding to the school’s core academic areas. 

Academic Program 

I . Students at the Downtown Academy Charter School will become clear and effective 
writers of the English language. 

I I . Students at the Downtown Academy Charter School will become proficient in and 
demonstrate continuous improvement of their reading skills. 

III. Students at the Downtown Academy Charter School will become proficient in and 
demonstrate continuous improvement of their math skills. 

IV. Students at the Downtown Academy Charter School will demonstrate mastery of critical 
knowledge and skills in the area of science. 

V. Students at the Downtown Academy Charter School will demonstrate mastery of critical 
knowledge and skills in the area of history. 
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The document makes a series of clear, precise statements about the school’s goals for its students. 
The Downtown Academy’s goals are clear enough that little if any explanation is necessary, making 
the document both measurable and transparent for all readers. 

^ For livery Goal, List One or More Objectives By Which it Will be Assessed 

Once a school has framed its goals, it then must describe how it will 
measure its progress and define what level of performance indicates 
that the school is meeting or exceeding performance expectations. It 
is not enough for a school to say that it will give students a certain 
assessment or use a certain technique to evaluate itself. Schools 
must indicate how they expect their students to perform on each 
measure if they are indeed making strong progress. If the school met 
its performance goals, what would students know and be able to do? 

How would they act? What would parents say about the program? 

Each goal must have at least one measure describing how it will be 
assessed. Some goals may have just one measure; others may have 
three, four, or more. Generally, schools should measure more extensively and accurately those goals 
that are most central to a school’s work and those goals that relate to complex ideas or results in 
order to ensure accuracy, objectivity, and reliability. The use of multiple measures and objectives to 
evaluate critical areas of student achievement is discussed later in the Guidelines and its appendices. 

^ Set a Umited Number of Goals 



Once a school has 
framed its goals , it then 
must describe how it 
will measure its 
progress and define 
what level of 
performance indicates 
that the school is 
meeting or exceeding 
performance 
expectations. 



An Accountability Plan is not designed to measure all of the work a charter school does. It is 

designed to measure the most critical areas of its performance, 
those areas that would inform a decision about whether to renew 
the school’s charter. Effective measurement and reporting can 
require a significant commitment of time and resources. Generally, 
it makes for a more powerful case if a school measures fewer things 
better than if it measures many things incompletely or superficially. 
Articulating a limited number of clear, critical, and carefully measured goals allows a school to make 
a more convincing case for its excellence and can act as a powerful tool in decision-making on the 
school level. 



An Accountability Plan 
measures the most 
critical areas of a 
school’s performance. 



y All Language Should Reflect the Goal of Measurability 



The Accountability Plan is an agreement between the school, the Department of Education, and 
members of the public served by the school. Precise 
language makes this agreement clear, while ambiguous 
language may lead to different interpretations depending 
on the reader. For instance, a school would be entirely 
justified in aspiring for its students to make “strong yearly 
progress” in their math skills, but it also must define what “strong progress” means and how it is 
identified if it does occur. In some cases, schools can provide this clarification through the effective 
use of the objectives for each goal. In other cases, clearer language may be necessary. 



Accountability Plans should 
avoid trying to assess what 
cannot be measured. 
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Similarly, Accountability Plans should avoid trying to assess what cannot be measured. The feelings, 
beliefs, and perceptions of individuals or groups of people, for example, are hard to measure 
reliably. For instance, a goal of helping students believe they can learn represents an attempt to 
measure what students believe about themselves and their capabilities. Students might say they 
believe in themselves in a survey, but do they do so according to their definition of belief or their 
teachers’? Learn what? To whose standards? In addition, the likelihood that anyone would answer 
“no” is low when asked, “Do you believe you can learn?” These same issues arise with the 
measurement of concepts such as “potential,” as in, “Students will achieve their full potential as 
writers.” Who, for example, determines a student’s potential and how? 

^ Measure Results as Opposed to Inputs 

Schools should resist the temptation to measure inputs such as plans, methods, and intentions. 

Some of the most commonly measured and discussed forms of school data, such as class size and 
professional development plans, are in fact inputs. Decisions around class size is a strategy, a 
practice followed by a school which may lead to increased performance, and does not belong in an 
Accountability Plan. Similarly, a goal focused on the number of newsletters sent out to parents each 
year shows very little about the amount of parent involvement in the school, the quality of that 
involvement, or the ultimate goal of that involvement. The focus of the Accountability Plan is on 
outcomes. 

Measuring outcomes rather than inputs is especially important in the area of academics where 
persistent or increased student achievement is the most compelling evidence of the success of an 
academic program. Strategies such as curriculum alignment or increase of after-school tutoring are 
vital to the program, but remain strategies to achieve a goal and not goals or objectives in 
themselves. 

As is consistent with the charter school initiative, each school will use different means to reach their 
goals. While a school should clearly articulate these “strategies for attainment” on the school level 
and in the school’s Annual Report, these should be left out of the Accountability Plan. The three 
issues guiding charter school accountability are focused on outcomes, not strategies implemented. 

^ Aim for clarity in content 

Schools should ensure that their Accountability Plan is clear and understandable by any reader. 
Charter schools, in seeking to provide parents with information about the education of their 
children and to demonstrate success of their program to the Board of Education, must provide clear 
and useful information suitable to serve as the basis for sound decision-making. This imperative 
particularly applies to the Accountability Plan and to the Annual Report. Any reader should be able 
to read either of these documents and understand exactly what a school has set out to do and 
whether they have accomplished it. 

^ Beware of Perverse Incentives 

In writing Accountability Plans, schools should avoid goals that could create an incentive to make 
decisions ultimately not in the best interests of the organization. For instance, a measure focused on 
teacher retention, if not phrased correctly, could encourage keeping poorly performing teachers in 
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order to maintain retention rates. Precise language can help avoid this situation. In this example, 
drawing a distinction between teachers who leave voluntarily and those who are asked to leave by 
the school might be one solution. Of course, if a situation arises where a decision made is best for 
the school but hurts the school’s level of achievement of a goal, a school has the opportunity to 
discuss this matter in its Annual Report. 

^ Prepare a Plan that Reflects the Timeline for Charter Renewal 

In writing Accountability Plans, schools need to remember they must have collected sufficient 
evidence of success by the time of renewal. In setting goals, it is important that these timelines 
inherent in the charter are taken into account. 

Between the spring of the third year and the end of the fourth year of their charter, charter schools 
must present their case for renewal in their Application for Renewal. This timeline affects the type 
of goals or objectives chosen in the Accountability Plan. For instance, a college preparatory charter 
school might propose to measure itself according to how many of its graduates enrolled at what 
colleges and the performance of those students in college. If the school opened its doors with only 
sixth graders, however, it would not be able to present any evidence that it had achieved its goal by 
the time it faced its first renewal because its oldest students would be only in the 1 1 th grade at that 
point. 

A school opening with Kindergarten and growing one grade per year until it was K-6 would face a 
series of problems at renewal if it relied on only the 4 th grade Massachusetts Comprehensive 
Assessment System (MCAS) to provide evidence of student achievement. In this case, the school’s 
first class of 4 th graders would take the MCAS in the spring of the fifth year of the charter, leaving 
the school with no data to present at the time of renewal. Even if the school’s first students took 
the MCAS during its 4 th year, MCAS scores would not be reported in time for the school’s fourth 
year Annual Report or for the start of the renewal process. 

In fact, if the MCAS were the only measure the school used and its first students took the test for 
the first time in the third year of the charter, the school would still face the challenge of trying to 
convince evaluators to draw conclusions from a single piece of data, one administration of the test. 
Moreover, it would limit its ability to convince parents, the general 
public, and potential funders that it was making strong progress 
during each year of its operations. In addition, a school would have 
no way to evaluate its progress in meeting goals they have deemed 
most important - a key element of self-evaluation. 

In short, as interested parties increasingly look for reliable, consistent 
evidence of performance throughout a school’s life cycle, charter 
schools come to realize that they are always accountable. A strong 
Accountability Plan ensures that a school will have reliable, objective 
data regarding its performance at every stage of its existence. 
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Academic Program Goals 



A school’s primary priority is the academic achievement of its students. Therefore, it is imperative 
that charter schools provide solid and convincing data regarding the success of their academic 
programs. 

^ Use Multiple Measures and Objectives 

As public schools, charter schools in Massachusetts are required to administer MCAS tests. The 
importance of this test to charter schools in terms of measuring effectiveness is underscored by 
NCLB. The Massachusetts Department of Education uses the MCAS as the main gauge of whether 
a school is making adequate academic progress under NCLB. 

Nonetheless, schools are encouraged to use multiple tools to demonstrate the effectiveness of all 
aspects of the school’s program. For example, a school in which students achieved extremely high 
MCAS scores in the 4 th grade might be accused of “creaming” the best students from its local 
district or districts. In order to contravene this argument, the school might use another measure to 
show that its students had improved significantly over time against their own earlier scores, 
something that the MCAS cannot yet do. Further, the school might also wish to demonstrate that 
its students had achieved solid mastery in topics based on its unique school design. 

The solution to the challenge posed by the use of only one assessment tool is to use assessments 
strategically and in combination. This strategy 
allows for the analysis of a variety of data that can 
show the depth of academic success. To 
adequately measure an important or complex goal 
a school should plan to measure the goal in several 
different ways and with several different tools. 

Different kinds of measures serve different purposes and allow for different, often mutually 
reinforcing, arguments. See Appendix I for a discussion of various methods of assessing academic 
progress. In addition, schools should be aware of what are considered statistically reliable methods 
of reporting data. Refer to Appendix II for more discussion on this topic. 

^ Adequate Yearly Progress, NCLB, and Accountability Plans 

NCLB requires that all students reach proficiency on state standards by 2013-2014. AYP represents 
the minimum level of improvement that schools must achieve each year in order to meet this goal. 2 
As public schools, charter schools are accountable for all the objectives set by the state 
accountability system. For renewal of a charter, MCAS results and AYP determinations are 
important pieces of evidence that is collected and analyzed to determine whether a charter school 
has met the criteria for renewal at the end of its five-year charter. MCAS goals in each 
accountability plan should be consistent with the overall goals of making AYP. While AYP 
determinations do not replace the accountability system that has been established for Massachusetts 
charter schools, charter schools should not overlook the importance of AYP determinations, both in 
their yearly practice as well as in their Accountability Plan. Objectives indicating successful overall 
school performance on the MCAS must be consistent with AYP goals. 

2 For more information regarding NCLB and charter schools, refer to Charter School Technical Advisory 03-2 : The 
Impact of NCLB on Massachusetts Charter Schools 
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Organizational Viability Goals 



Certain criteria can help demonstrate a charter school's organizational viability. These include 
demand for the school’s services, sound finances, as well as capable governance and management. 

In general, the school should strive to demonstrate that it is a sustainable organization and likely to 
continue operating in an effective and responsive manner. 

^ Measure the Market 

One potential threat to an organization’s viability is a lack of demand for the organization’s services. 
If spaces remain unfilled, a school is likely to face financial difficulties in due time. Even before 
such a crisis occurs, the Commonwealth might question why it should commit scarce resources to 
provide an option few parents seem inclined to choose. Therefore, schools should seek to 
demonstrate that parents choose and persist with the school. 

Schools should set enrollment goals and consistently report aggregate demand figures, such as the 
average percentage of seats filled during the year, the percentage of students who return each year, 
and the ratio of applications to available seats. Schools may also wish to provide further analysis of 
enrollment data, such as setting goals for and 
assessing the reasons families give for their 
decisions to leave. Do transportation problems 
drive families away? Do they leave because the 
academics are too difficult or too easy? 

Depending on the nature of the responses, 
such data can suggest complex trends and present mixed signals. Like many aspects of goal-setting 
and reporting, demand data may require further investigation and analysis by schools. 

Additional data that can demonstrate demand include waiting lists and parent satisfaction. If such 
information is used to show organizational success in an Accountability Plan, it needs to be done in 
a reliable and consistent manner. In the case of waiting lists, the manner in which the lists are kept 
and recorded should be consistent throughout the years of the charter and data should be tracked 
and reported accordingly. Parent satisfaction is most clearly gauged through the number of students 
who come and stay at a charter school, and additional information can be measured through tools 
such as surveys. Please see Appendix III for more information regarding the use of surveys. 

^ Track the Money 

Other potential threats to a school’s organizational viability come in the form of financial issues such 
as a lack of adequate financial controls or poor financial decision making. Schools should commit to 
having sound financial practices and effective decision making procedures and to implementing a 
system capable of demonstrating that soundness. Each school must demonstrate financial 
transparency by submitting the required independent annual audit for each fiscal year. A school 
must also demonstrate its overall soundness by submitting actual and proposed budget for every 
fiscal year and an annual balance sheet demonstrating an adequate availability of funds to sustain 
operations in each Annual Report. 

It is not enough for schools to say that these documents will be submitted. As with goals in the 
academic program area, schools must define the level of performance that indicates the effectiveness 
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of the school’s systems and program. Evidence of a balanced budget, ability to raise funds, and 
unqualified opinions on annual audits are good indicators of financial stability. For example, one of 
the Downtown Academy Charter School’s goals might be to “demonstrate the sustainability and 
stability of the school through the careful use of financial resources.” The measure of this goal 
might state that “actual and proposed budgets for each fiscal year will demonstrate effective financial 
decision-making suitable to sustain key programs and a record of academic effectiveness. Further, a 
yearly independent audit with positive findings will give clear evidence of sound financial practices.” 

In planning financial operation and reporting systems, schools should remember that financial 
mismanagement has been by far the most common cause of charter revocations and non-renewals 
across the country. Charter schools are public entities and therefore must ensure that their finances 
are entirely transparent and that their practices above any possibility of reproach. 

^ A Viable School Must Demonstrate Sound Governance and Management 

The Board of Tmstees of a charter school, as the holder of the charter, plays a more significant role 
than does the Board of almost any other organization. A charter school board is equivalent to the 
school board in a small district and as such represents the first line of accountability. When and if 
issues arise in a school, the Department of Education will often rely on the Board of a school to 
devise solutions or conduct investigations. This is particularly important because charter schools are 
independent organizations. Even though the Charter School Office may identify concerns in school 
evaluations, it generally does not propose specific solutions, and the task of resolving problems falls 
to the Board of Trustees. 

If evidence exists that a school’s Board of Tmstees is not operating effectively, is not carrying out its 
duties, or appears less than fully capable of carrying out those duties effectively, then the school’s 
viability is in question. Each school must determine how best to provide evidence of its own sound 
and effective governance. Many schools have found an analysis and discussion of the minutes of 
their own Board meetings or of key Board decisions in a given year to an be effective tool. Some 
Boards have chosen to undergo outside evaluations of their practices to help determine their 
strengths and weaknesses and affect their practice. 

Some other key organizational issues that can be measured include the effectiveness of the school 
leader and teachers. A school might wish to measure the number of pre-determined goals that are 
met each year by the leader or how many of their teachers reach a certain level of success on their 
evaluation based on a consistent and clearly defined mbric. As mentioned earlier, it is important to 
avoid “perverse incentives” and keep in mind that the validity of objectives in this section are made 
stronger if they are corroborated or evaluated by a qualified external body. 

^ Get to SLesults 

In the Organizational Viability section of the Accountability Plan, as in all others, schools should 
recall their commitment to be held accountable for results. As discussed earlier, this commitment 
often poses particular challenges in the final sections of the Plan, regarding Organizational Viability 
and Faithfulness to the Charter. Schools, however, can and must create the objectives that best 
prove their effectiveness in these areas. 
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There are times when a school may choose to include a process goal or measurement in the sections 
regarding Organizational Viability or Faithfulness to Charter. Such process goals, which focus on 
how something will be done, are often based on what has been proven to be strong organizational 
practice. For example, a goal regarding Board practice could include a measure that each year it will 
participate in a rigorous and systematized self-evaluation that includes input from the school 
community and whose results are made public. While the accomplishment of this measurement 
would not prove that the Board is a strong functioning body, it would provide important 
information regarding Board effectiveness. If such process goals or measurements are used, the 
school should be able to articulate why that specific measure was chosen by the school community. 
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Faithfulness to Charter Goals 



The Accountability Plan, as the primary tool by which schools demonstrate their progress, should 
include several assessments of the school’s faithfulness to the unique concepts and programs it has 
proposed in its charter application. In designing such assessments, schools should remember that 
Accountability Plans are designed only to measure the most important elements of a school’s 
program and that they are designed and intended to measure results not plans. 

Typically, a school would want to measure its effectiveness in implementing the most important 
aspects of its program. For example, a school with a bilingual program would want to measure its 
students’ facility in their second language. A school with a strong emphasis on culture and 
community might wish to assess 
the school climate. A school that 
intended to serve a particular 
population of students (at-risk 
students, for example) would 
want to track the progress of that 
group independent of the student 
body as a whole. A technology-focused school would want to measure its students’ ability to apply 
computer skills. 

In understanding how to best measure key elements of the charter, it may be helpful to use the 
technology-focused school as an example. The school’s first inclination might be to set goals for 
and assess the ratio of computers to students or the number of hours a day when students use 
computers. While such aspects are important elements of the program, they indicate little about 
what students have learned about and are able to do with technology. 

A better approach might be to develop a list of key technology skills that each student in the school 
should master by the end of first, fourth, and sixth grade. The faculty might constmct a rubric 
describing how well students should be able to use those skills to have satisfactorily demonstrated 
mastery. The school might then plan an assessment to take place at the end of each of those years 
that evaluates students’ ability to perform the skills in question. The school would then set goals for 
student performance on the assessment and have that rubric available for any outside reviewers to 
see. 

Effective measurement like this requires creativity in focusing on outcomes rather than inputs. In 
many cases, schools may find that tools do not yet exist to measure results in the areas they wish to 
evaluate. In such cases, schools may have to develop measurement tools themselves. Because 
schools, charter and district alike, throughout Massachusetts often seek to measure the results of 
similar areas of their programs, developing innovative tools to measure performance accurately and 
effectively is an ideal area in which schools can develop partnerships and/ or disseminate their best 
practices. 



The Accountability Plan, as the primary tool by 
which schools demonstrate their progress, should 
include several assessments of the school’s 
faithfulness to some of the unique concepts and 
programs it has proposed in its charter application. 
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Drafting, Approval, and Reporting 



^ Deadlines and Process 

The timeline for the production of Accountability Plans is designed to encourage schools to begin 
gathering data early in the process of implementing their school design. At the same time, it allows 
schools to develop their goals and objectives with the input of those involved in implementing the 
programs covered by the Plan. The final version, approved by the school’s Board of Tmstees, is due 
on August 1 before the beginning of the second year of operation. For schools entering the second 
term of their charters. Accountability Plans are due at the time of the renewal application. 

A final Accountability Plan involves on-going discussion within the school community as well as 
between the school and the Charter School Office. Frequently, finalizing a Plan requires multiple 
drafts. Because the Accountability Plan sets the standards by which the school will seek to 
demonstrate its effectiveness, the Department may require or recommend additional clarity on 
various terms and objectives contained in a school’s Plan. Schools should understand that the 
Department, in approving Plans, reserves the right to do so with qualification. If a school includes 
language or objectives that the Department feels are inadequate in some way, whether they are 
unclear or not suitably objective or rigorous, it is possible that even successful or apparently 
successful achievement of the goals in the Plan as written might not result in the Department 
determining the school is a success. 

> Rt porting for Charter Schools Means Analysing Data More Than Tilling in Planks 

One of the most challenging aspects of charter school accountability is the change in the type of 
reporting that school leaders must do. Rather than merely plugging in data in provided forms, the 
accountability process for charter schools asks that each school analyze the information and 
construct an argument that is worthy of renewal based upon that information. 

The reality of data is that it has little meaning until it is analyzed, interpreted, and explained. Public 
officials, parents, the media and others continually use school data to draw conclusions, with varying 
degrees of accuracy and sometimes with little input from or understanding of the school. In 

Massachusetts, the accountability system under which 
charter schools operate provides schools with an 
opportunity to present not only data but also an 
interpretation and analysis of that data. Because others 
will inevitably draw conclusions from a charter school’s 
data, the opportunity to present its deeper analysis is a 
powerful tool for a school. For instance, the 
Department has MCAS scores for every school but it 
asks charter schools to report these data as part of the Annual Report, allowing the school to 
disaggregate and to interpret its scores to give a broader picture regarding student achievement. 
School personnel should keep the importance of analysis in mind as they write their Accountability 
Plan and their Annual Reports. 



In Massachusetts, the accountability 
system under which charter schools 
operate provides schools with an 
opportunity to present not only data 
but an interpretation and analysis of 
that data. 
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P More Than Compliance 

The primary purpose of a school’s Accountability Plan and the Annual Reports is to prepare and 
present a strong case for a school’s academic success, organizational viability, and faithfulness to the 
terms of the charter for an external audience. A well-written and tracked accountability plan is also 
an invaluable tool for a school’s internal use. The creation of the Plan can aid in defining for all 
stakeholders clear and focused goals, which then drive decision-making and practice - a hallmark of 
good organizations. The resulting data analysis and reporting on Accountability Plan goals may 
affect decisions regarding finances, 
curriculum, and other programmatic 
aspects. 

Schools that use their Accountability 
Plans as guiding documents are able to 
answer external reviewers’ questions 
about major school priorities and the 
extent to which they are being met. Schools that have strong Accountability Plans are also more 
likely to have implemented systems to track data efficiently and consistently and, therefore, are 
better prepared to write their Annual Reports and renewal application. 

^ Making Changes 

Though a school should write its Accountability Plan with an eye towards permanence or at least 
longevity, it is sometimes necessary to revise the document during the term of the charter. In such 
cases, schools should amend their Accountability Plans well in advance of renewal with sufficient 
time to collect data to provide evidence for new objectives. All changes must be approved by the 
Department of Education’s Charter School Office in order to take effect and a school is therefore 
governed by the most recent approved Accountability Plan on file with the Charter School Office. 

Conclusion 

Charter schools are, by definition, exceptional institutions. Charter schools accept the challenges 
that face all public schools and embrace a unique and demanding burden of proof in the 
accountability inherent in a five-year charter. Charter schools are built on the philosophy that 
success is possible for all children. In writing an Accountability Plan and in reporting its progress 
against it, a school embraces a commitment to both success and transparency. The Accountability 
Plan allows a school to sets goals that reflect its uniqueness and autonomy while giving substance to 
a school’s commitment to parents and citizens. 



The primary purpose of a school’s 
Accountability Plan and the Annual Reports is 
to prepare and present a strong case for a 
school’s academic success, organizational 
viability, and faithfulness to the terms of the 
charter for an external audience. 
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Appendix I: 

Methods and Tools for Measuring Student Achievement 



A school can use different assessment tools and different data-sorting methods to present overall 
student achievement. These Guidelines will discuss three main types of assessment tools ’ to measure 
student achievement: criterion-referenced, norm-referenced, and internally developed assessments. 
Data from such sources can be used and reported in a variety of ways to gauge and report student 
achievement: absolute, value-added, and comparative measures of achievement. These tools and 
methods work together to demonstrate measurable progress of student academic success. 

Assessment Tools 

r* Criterion-Referenced Tests 

Criterion-referenced tests are based on a fixed standard of achievement. They rate students’ 
performance not in relation to the achievement of other students but in relation to what experts and 
authorities believe an educated student should know and be able to do. The levels of achievement 
are determined prior to the administration of the test and remain the same no matter how students 
do. The MCAS is such an assessment. Hypothetically, every student in Massachusetts could pass or 
fail the exam and the required passing score would remain the same. Additional examples of 
criterion-referenced assessments are those included in programs such as the International 
Baccalaureate or Core Knowledge. 

^ Norm-Referenced Tests 

Norm-referenced tests are some of the more commonly used tests, such as the Stanford-9 or 10, the 
Iowa Test of Basic Skills, and the Terra Nova. These tests are usually multiple-choice and measure 
the student’s achievement against a standard established by a norm group. Individual student 
performance is compared to their peers nationally. These tests are particularly helpful in providing 
measures of growth over time. It should be noted that most normed, standardized tests measure a 
limited array of basic skills and are most effective in assessing progress in skill-intensive disciplines 
like math and reading. Because these tests are often significantly out of alignment with state learning 
standards in social studies and science, some schools find them less effective in measuring progress 
in these disciplines. 

^ Internal Assessment Sjstems 

Many schools wish to include objectives in their Accountability Plans that are based upon a system 
they have designed themselves. Such internal assessments can often provide rich and compelling 
data that is beyond the scope of externally designed measurement tools. These can take many 
forms, such as portfolios, juried assessments, project grades, and essays. It is important, however, 
that such an assessment system is designed and implemented in a way that is reliable and valid. A 
school should keep in mind the following mles in creating such a system: 



3 The Guidelines only touch briefly on the types of assessments that schools can use to reliably measure student 
achievement. There are many different assessments - both external and internal - and each school should be aware 
of the uses and designs of such assessments and choose those that best fit its program. 
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• Internal assessments should be scored according to specific standards using consistent protocols or rubrics. To 

collect meaningful data through an internal assessment, a school must ensure that such 
assessments are scored consistently throughout the school by each individual involved. Pre- 
determined rubrics are key to such a system, as are clear protocols for implementation and 
scoring of such assessments. 

• Internal assessment systems should be vetted and validated. If a school is using a common rubric and 
scoring protocol, it must ensure that these tools are being used by the entire faculty or group of 
graders in the same manner. How this is done varies from school to school. Some schools may 
use a process of gathering teachers or an internally created panel to look at student work and 
corresponding grades to ensure consistency. Some schools have their system vetted by experts 
outside the school community or have external reviewers grade the assessments from which 
important data will come. Some schools may choose to do this for their whole grading system 
and others may focus this type of attention only on key assessments used in their Accountability 
Plan to show student performance. Whichever method is chosen, it is important that schools 
make sure that this oversight is an integral part of their system. 

• Internal assessments should assess areas of knowledge beyond the scope of other measurement tools. Internal 
assessments in Accountability Plans serve little purpose if they fail to get beyond basic skills 
more easily assessed by standardized measures. Such internal assessments are most compelling 
when they show that students have mastered complex, “higher-order” skills. 

As an example of an internal assessment used in an Accountability Plan, a school might augment 
standardized measures of its students’ writing abilities with an assessment that asks students to write 
a five-paragraph essay analyzing a novel. Such an assessment would provide compelling data if the 
results were scored by an internal committee or qualified, independent experts, such as a panel of 
local English teachers and community college professors. The school would also want to document 
that its panelists had looked for the mastery of specific skills and knowledge and were not grading 
students on the physical appearance of their essays or on undefined criteria. To address this 
challenge, a school would need to ensure that panelists adhered to a consistent mbric developed by 
the school and available to outside reviewers 

Effectively designing and evaluating a school’s own internal assessment system requires a significant 
amount of work. While schools may choose to rely on externally validated standardized tests, some 
schools’ internal assessment systems are integral to their educational approach and therefore key to 
their measure of student progress. If used for external reporting, schools must be prepared to 
report results in a way that demonstrates consistent learning and achievement among entire cohorts 
of students, not merely among selected individuals. A school-developed assessment system that can 
only demonstrate the progress of individual students should not be included in an Accountability 
Plan because conclusions cannot be drawn about the school’s effect on the majority of its children. 

Methods of Reporting 

^ Absolute Measure of Student Achievement 

To measure success in reaching an Accountability Plan goal, some schools may choose to use 
absolute measures of student achievement. Such goals set an ultimate goal of where a school wants 
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its students to be on a certain assessment - such as 80% of students scoring proficient or above on 
the MCAS tests, or all students scoring above the 50 th percentile on a norm-referenced test. These 
particular objectives are not focused on growth but rather on the ultimate level of achievement 
expected by the school. An absolute measure of achievement via a criterion-referenced assessment 
provides a clear reference to the ultimate goal, sets high standards, and, because the MCAS is 
required of all schools in the Commonwealth, should be routine for charter schools. In fact, the 
NCLB goal of 100% of students reaching proficiency by 2014 is an example of an absolute measure 
for which each school must be accountable. 

^ The Value-added Measure of Student Achievement 

Value-added measures chart student progress over time on a particular assessment, comparing 
students’ scores to their own previous scores. Looking at data in this way can provide an indication 
of how much students have progressed academically within a given time frame. When doing this 
type of analysis, schools should pay particular attention to tracking intact cohorts of students. The 
Guidelines discuss this further in Appendix II. 

Because a value-added measure compares students to themselves, it is the most effective tool for 
eliminating demographic issues such as the socio-economic status of students. When student scores 
are aggregated, a value-added measure allows administrators, regulators, and parents to see how 
much a school has added to its students’ performance. The argument presented by such measures 
can contradict accusations that a charter school is “creaming” students, as well as ensure that charter 
schools are meeting their obligations to students of all academic levels. In the case of students who 
arrive performing poorly and perhaps several years behind their peers, a value-added measure allows 
a school to demonstrate evidence of learning that may not yet show up on a criterion referenced 
assessment that is tied to specific grade level learning standards. Without a value-added measure, the 
progress of an eighth grade student whose math skills improved from the 4 th to the 7 th grade level 
might be missed. At the other extreme, a school that inherited a population of students who were 
already performing consistently above grade level might not be able to demonstrate clearly the 
school’s contribution to the students’ performance without value-added data. 

Most schools currently generate effective value-added data from standardized, nationally-normed 
assessments. These tests can be given to students each year and are specifically designed to allow for 
year to year comparison. Value-added measurement can also be achieved using an internal 
assessment system provided that it is designed to collect such data reliably. It is, however more 
difficult to collect this data using an internal assessment system than when using a norm-referenced 
test externally scored. Ultimately, when the MCAS has been implemented in all intended grade 
levels 4 , it may allow schools to generate some comparative data across years, although not in the 
same way as norm-referenced tests. Because this is the case, schools should be prepared to gather 
value-added data through other assessments. 

^ The Comparative Measure of Student Achievement 

Choice is central to the charter school movement and comparative measures provide information 
that enables effective choice, particularly for parents who may be deciding on an appropriate school 



4 To comply with NCLB, the Department will be administering both Math and English Language Arts MCAS tests in 
grades three through eight and in grade ten starting in the spring of 2006. 
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for their children. Comparative measures also allow the charter authorizer to compare a charter 
school’s progress against the progress of similar populations of students and schools. 

There are several methods by which a school can establish a comparative measure of its 
effectiveness and its students’ progress. If a school draws its students almost exclusively from a 
single school district, for example, it might choose to compare its performance to the district’s 
performance on assessments that both entities use, such as the MCAS. Alternatively, a school might 
choose to compare its progress to the average of several specific schools from which it drew its 
students, with the average weighted to reflect the relative proportion of students that came from 
each school. A school might choose this technique in particular if the schools from which it drew 
its students were not reflective of the overall district performance level. A charter school might also 
compare itself to schools with similar demographics. Schools that draw their students from several 
districts also could compare their progress to each of the districts from which they drew a significant 
number of students or can compare their progress to a weighted average of those districts. 

^ Example: Hoiv Multiple Objectives Might Be Used to Evaluate Reading Skills 



Goal: Downtown Academy students will become proficient readers of the English language. 

Measures and Objectives: 

1) Eighty percent of students who have attended Downtown Academy for two or more years will 
score proficient or advanced on the MCAS English Language Arts test. 

2) The average national percentile ranking of each cohort of Downtown Academy students will 
increase by an average of three percentiles per year on the reading battery of the ITBS until the 
average national ranking of the cohort is 70%. 

3) Aggregate scores for students at Downtown Academy will place the school among the top 
25% of middle schools in the City School District on the MCAS English Language Arts test, as 
measured by the proficiency index. 

4) Ninth grade students at Downtown Academy will present an essay in which they compare 
and contrast character, plot, and theme in two classic novels. The essays will be presented to 
a panel of local professors and teachers. At least 70% of students will receive a rating of 
proficient or better from the panel on the attached rubric. 



In the example, the Downtown Academy Charter School used various measures to assess one of its 
most critical goals: to ensure that its students’ are proficient readers of English. The school used an 
absolute measure, the percentage of students who pass the MCAS; a value-added measure, the 
average improvement of students on the ITBS; a comparative measure, its MCAS scores versus the 
rest of the District, and an in-depth measure that is externally evaluated and scores students 
according to specific, appropriate criteria. Through the use of the multiple measures, the school is 
likely to present compelling and thorough evidence of its students’ proficiency in reading. 
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Appendix II: 

Statistically Sound Measurement 



When using norm-referenced tests, such as the Stanford-9 or the Iowa Test of Basic Skills, it is 
important to understand the formats testing companies use to report the students scores. 3 Each 
format has different meanings and different uses. 

^ Runv and Scaled Scores 

Data reports from norm-referenced tests describe the performance of students or a student as a 
“raw score” and “scaled score.” The raw score indicates the number, or average number, of 
questions answered correctly. The scaled score corrects the raw score by compensating for the 
relative difficulty of the specific version of the test. The scaled score is used to determine three 
other numbers discussed below that school leaders should understand. 

^ Percentile Rank and Normal Curve Equivalents 

When translated onto a normal curve, norm-referenced tests yield a score called the percentile rank. 
This score places each student on a curve of all test takers 6 . The score indicates the percentage of 
test takers that an individual or group has out-scored. That is, a student who scores at the 55 th 
percentile has out-performed 55 percent of the students in the nation at his or her grade level. 

Percentile rank scores are commonly used by educators, with good reason but not always correctly. 
Percentile ranks are intuitive, easily understood by parents, and provide a precise measure. For 
instance, a measure of a student’s percentile rank over several years that went from the 43 rd to the 
55 th percentile in the course of three years indicates that the student’s knowledge and skill level 
increased not only in the absolute sense but also relative to a typical population of his peers. This 
student passed 12 percent of the students over three years. Percentile rank scores, when tracked 
over time, reveal a great deal of information. 

Though they are both intuitive and responsive, percentile rank scores pose several problems. One 
significant problems is that scores of more than one student cannot be averaged together to 
compute a mean score for a classroom or a school without first being converted into another form. 
On a normal curve, an increase from one score to the next higher score on the curve does not 
necessarily indicate an equal increase in actual performance as would be indicated by a similar gain at 
another point on the curve. In other words, the “interval”, or distance between a score and the next 
higher score, at the 50 th percentile is different from the interval at the 90 th percentile. Because so 
many more students are grouped near the middle of the curve, moving from the 50 th to the 51 st 
percentile requires only a tiny increase in the number of questions answered correctly. With fewer 
students at the 90 th percentile, a student might have to answer several more questions correctly to 
show the same one percentile increase. 



5 Additional information and assistance on understanding the forms and use of standardized test scores is available 
from testing companies and other assessment resources. 

6 The performance of “all students” is extrapolated from the performance of sample groups. 
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This problem can be addressed by using a tool called the normal curve equivalent (NCE). Test 
makers report scores in NCE by performing additional calculations on the percentile rank scores to 
make them useful for averaging and comparing year to year. Like percentile rank scores, NCE 
scores are also reported on a 1 through 99 scale where a score of 50 also represents the mean score. 
An NCE score, however, doesn’t represent the same tiling as a percentile rank score does. A 
percentile rank score of 53 means that a student outscored 53 percent of her classmates nationwide. 
An NCE score of 53 is a derivation from another number. That’s a hard concept to discuss with 
parents when assessing their students’ or the school’s performance. Still, NCE scores can be safely 
averaged, aggregated, and used to calculate in myriad ways. What this means for schools writing 
Accountability Plans is that they should plan to either measure and report scores in NCE or, if they 
prefer the intuitiveness and “meaning” of percentile rank, to calculate the various means they require 
in NCE and then convert them into percentile rank as the very last step in the process. 

^ Grade Level Equivalents 

One other score format bears mentioning, as it is one of the most commonly used score formats, 
although the most misleading. Grade level equivalent scores report a score for each student 
indicating a grade level and a number of months reflected by their performance on a given test. A 
grade level equivalent of 5.2 indicates performance akin to that of a fifth grader during the second 
month of the school year. While this appears to be a clear and useful measure and one that is 
intuitive for both parents and teachers, statisticians indicate that they can be misleading if not 
inaccurate. For example, assume that a student who scores at grade level 5.2 was a fourth grader. A 
parent might be inclined to conclude that her daughter was scoring at the fifth grade level and 
should be assigned fifth grade level work. In fact, the measure indicates the performance level one 
would expect of a typical fifth grader on fourth grade level work , a very indirect measure. For this 
reason, test specialists consider grade level equivalents with far more circumspection than they do 
most other data and are generally reluctant to use them. 

In addition, grade level equivalent scores, like percentile rank scores, cannot be used accurately to 
calculate the averages or means of more than one students’ scores. Thus while schools may find 
them useful for discussing the scores of a single student with individual parents, grade level 
equivalents are not an effective or appropriate tool for reporting performance data to public officials 
or to the general public in the Accountability Plan or Annual Report. 

'E Tracking Cohorts 

When looking at assessment data over time, the most statistically accurate comparisons from year- 
to-year are made at by tracking cohorts of students. For purposes of the Accountability Plan, a 
cohort is an intact group of students that does not change over time. For example, a school serving 
fifth through eight graders gives every student the Stanford-9 upon entrance to the school and 
administers the test each consecutive spring until students graduate from the eighth grade. Its class 
of 2003 took its initial test in the fall of 1999 and then each subsequent year until its final 
administration in spring of 2003. In attempting to look at the improvement in performance over 
time of its students, this school might be tempted to compare the aggregated NCE scores from the 
initial fall administration in 1999 to each subsequent test for the class of 2003. While this appears to 
be looking at performance longitudinally, it does not take into consideration students who left or 
entered the class of 2003 during the past four years. Therefore, the scores used are not comparable 
because they have not been controlled for varying enrollment. A more statistically accurate analysis 
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would include only those students who have been at the school for every, eliminating the scores of 
those students who were not part of the initial administration. 

Tracking cohorts is not only more statistically accurate, it also allows a school to make the argument 
that the longer a student stays at the school, the better they perform. Such an analysis should 
control for scores of students who enrolled only weeks prior to the test administration, which could 
skew the aggregate score of the grade. 

Tracking intact cohorts can also be used when creating absolute measures for other assessments 
such as MCAS. While each school’s MCAS scores will be reported as an aggregate of all current 
students, and these scores are the basis of determining AYP, a school can provide additional 
information on how those students who have been at the school for a specified amount of time 
perform. This gives evaluators information regarding the effect the school has had on achievement 
for those students who have benefited from the program a longer period of time. 

^ Small Sample Si%e 

Another problem can arise when schools compute averages or means, particularly on nationally- 
normed tests, using the test scores of a very small group of students. Generally, the greater the 
number of students included in calculating a mean, the less likely that a few anomalous scores will 
skew the results. However, schools in general, and charter schools even more so, are small 
institutions that generate small pools of data. Statisticians indicate that averages including fewer 
than 20 test takers are less accurate. Schools should be aware of the inherent challenges of testing 
on a small scale, even while they remember that the burden of proof is theirs. It is incumbent upon 
each school to find the best available solution to its particular circumstances. Schools should 
approach this challenge by working in consultation with the Charter School Office and with the 
understanding that irrespective of the circumstances, test results in general, and the MCAS in 
particular, are both viable and necessary tools for the evaluation of all charter schools. 
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Appendix III: 
Use of Surveys 



A well-developed survey takes time, but once designed and vetted it can be a highly effective 
measure of the school’s relationship with stakeholders. In using surveys, schools should be aware of 
several specific challenges. 

Surveys are most effective when carefully and specifically worded to measure a school’s effect on its 
students, with a clear focus on results. Consider the difference between a survey goal stating 
“eighty percent of parents will agree that their students understand and appreciate art” and a goal 
stating that “eighty percent of parents will agree with the statement, ‘The school has increased my 
child’s understanding and appreciation of the visual arts.’” Parents could easily assume the first 
question asks for an assessment of their children’s affinity for the arts. Clearly, most parents love 
their children and are inclined to believe that they appreciate things, and parents are more inclined to 
feel more favorably about their own children than they are about even the finest school. Therefore, 
such a question is likely to skew results to the positive. More importantly, even if parents are correct 
in their beliefs about their children’s appreciation for the arts, the point is not whether students appreciate 
the arts hut whether the school has increased their capacity to do so, something the second question asks about 
specifically and directly. 

Surveys are also most effective when questions are highly specific. The second goal in the example 
above is stronger because it includes a direct quotation of the question asked of parents. In another 
example, a Plan might suggest a measure that “eighty percent of parents will report, in an annual 
survey, that they are satisfied or highly satisfied with the school.” Compare this to a measure stating 
that “eighty percent of parents will report that they agree or strongly agree that ‘teachers in the 
school are effective and set high standards for students.’” The first measure gives no indication of 
the criteria used to determine satisfaction. Perhaps parents are satisfied because lunches are tasty or 
because the school is right on their route to work in the morning, criteria that are nearly irrelevant in 
terms of determining renewal. The second measure specifies the criteria for parent satisfaction and 
directly addresses a specific outcome of critical importance. 

A few other aspects of the use of surveys affect the reliability of the information gathered. The first 
is the rate of return. Schools may very well find that a survey of particular stakeholders only yielded 
a relatively small percentage of the returned surveys. If only fifteen percent of the parents return a 
survey, those results cannot be considered to reflect the general opinions of the parents - only of 
that fifteen percent. In addition, the way in which surveys are administered, collected, and tabulated 
has a large effect on how reliable the information is. To truly gather opinions from stakeholders, 
anonymity must be assured and the data must be collected and aggregated correctly. Many schools 
address this aspect by hiring an outside survey firm to send and collect survey data. There are other 
ways in which this can happen — through the internet, with certain computer programs, or with a 
well-developed paper system - but schools must be cognizant of these challenges when using 
surveys. 
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